

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>OSD Service &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/underscore.js"></script>
        <script src="../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="RGW Service" href="../rgw/" />
    <link rel="prev" title="MGR Service" href="../mgr/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../">Cephadm</a> &raquo;</li>
        
          <li><a href="../">Service Management</a> &raquo;</li>
        
      <li>OSD Service</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../_sources/cephadm/services/osd.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../">
          

          
            
            <img src="../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">Cephadm</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../compatibility/">Compatibility and Stability</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../install/">部署个全新的 Ceph 集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../adoption/">现有集群切换到 cephadm</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../host-management/">Host Management</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">Service Management</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../mon/">MON Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mgr/">MGR Service</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">OSD Service</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#list-devices">List Devices</a></li>
<li class="toctree-l4"><a class="reference internal" href="#deploy-osds">Deploy OSDs</a></li>
<li class="toctree-l4"><a class="reference internal" href="#remove-an-osd">Remove an OSD</a></li>
<li class="toctree-l4"><a class="reference internal" href="#automatically-tuning-osd-memory">Automatically tuning OSD memory</a></li>
<li class="toctree-l4"><a class="reference internal" href="#advanced-osd-service-specifications">Advanced OSD Service Specifications</a></li>
<li class="toctree-l4"><a class="reference internal" href="#examples">Examples</a></li>
<li class="toctree-l4"><a class="reference internal" href="#activate-existing-osds">Activate existing OSDs</a></li>
<li class="toctree-l4"><a class="reference internal" href="#futher-reading">Futher Reading</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../rgw/">RGW Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mds/">MDS Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../nfs/">NFS Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../iscsi/">iSCSI Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../custom-container/">Custom Container Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring/">Monitoring Services</a></li>
<li class="toctree-l3"><a class="reference internal" href="../snmp-gateway/">SNMP Gateway Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#service-status">Service Status</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#daemon-status">Daemon Status</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#service-specification">Service Specification</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#daemon-placement">Daemon Placement</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#removing-a-service">Removing a Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#disabling-automatic-deployment-of-daemons">Disabling automatic deployment of daemons</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../upgrade/">Ceph 的升级</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../operations/">Cephadm operations</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../client-setup/">Client Setup</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../troubleshooting/">Troubleshooting</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../dev/cephadm/">Cephadm Feature Planning</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="osd-service">
<h1>OSD Service<a class="headerlink" href="#osd-service" title="Permalink to this headline">¶</a></h1>
<div class="section" id="list-devices">
<h2>List Devices<a class="headerlink" href="#list-devices" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">ceph-volume</span></code> scans each host in the cluster from time to time in order
to determine which devices are present and whether they are eligible to be
used as OSDs.</p>
<p>To print a list of devices discovered by <code class="docutils literal notranslate"><span class="pre">cephadm</span></code>, run this command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "# ";
}
</style><span class="prompt1">ceph orch device ls <span class="o">[</span>--hostname<span class="o">=</span>...<span class="o">]</span> <span class="o">[</span>--wide<span class="o">]</span> <span class="o">[</span>--refresh<span class="o">]</span></span>
</pre></div></div><p>Example</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Hostname</span>  <span class="n">Path</span>      <span class="n">Type</span>  <span class="n">Serial</span>              <span class="n">Size</span>   <span class="n">Health</span>   <span class="n">Ident</span>  <span class="n">Fault</span>  <span class="n">Available</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">P0A0YFFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A08WFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdd</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A07DFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sde</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">P0A0QDFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A033FRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A05XFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sde</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0ANFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdf</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A06EFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">03</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0OGFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">03</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0P7FRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">03</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdd</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0O7FRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
</pre></div>
</div>
<p>Using the <code class="docutils literal notranslate"><span class="pre">--wide</span></code> option provides all details relating to the device,
including any reasons that the device might not be eligible for use as an OSD.</p>
<p>In the above example you can see fields named “Health”, “Ident”, and “Fault”.
This information is provided by integration with <a class="reference external" href="https://github.com/libstorage/libstoragemgmt">libstoragemgmt</a>. By default,
this integration is disabled (because <a class="reference external" href="https://github.com/libstorage/libstoragemgmt">libstoragemgmt</a> may not be 100%
compatible with your hardware).  To make <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> include these fields,
enable cephadm’s “enhanced device scan” option as follows;</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph config <span class="nb">set</span> mgr mgr/cephadm/device_enhanced_scan <span class="nb">true</span></span>
</pre></div></div><div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Although the libstoragemgmt library performs standard SCSI inquiry calls,
there is no guarantee that your firmware fully implements these standards.
This can lead to erratic behaviour and even bus resets on some older
hardware. It is therefore recommended that, before enabling this feature,
you test your hardware’s compatibility with libstoragemgmt first to avoid
unplanned interruptions to services.</p>
<p>There are a number of ways to test compatibility, but the simplest may be
to use the cephadm shell to call libstoragemgmt directly - <code class="docutils literal notranslate"><span class="pre">cephadm</span> <span class="pre">shell</span>
<span class="pre">lsmcli</span> <span class="pre">ldl</span></code>. If your hardware is supported you should see something like
this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Path</span>     <span class="o">|</span> <span class="n">SCSI</span> <span class="n">VPD</span> <span class="mh">0x83</span>    <span class="o">|</span> <span class="n">Link</span> <span class="n">Type</span> <span class="o">|</span> <span class="n">Serial</span> <span class="n">Number</span>      <span class="o">|</span> <span class="n">Health</span> <span class="n">Status</span>
<span class="o">----------------------------------------------------------------------------</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda</span> <span class="o">|</span> <span class="mi">50000396082</span><span class="n">ba631</span> <span class="o">|</span> <span class="n">SAS</span>       <span class="o">|</span> <span class="mi">15</span><span class="n">P0A0R0FRD6</span>       <span class="o">|</span> <span class="n">Good</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span> <span class="o">|</span> <span class="mi">50000396082</span><span class="n">bbbf9</span> <span class="o">|</span> <span class="n">SAS</span>       <span class="o">|</span> <span class="mi">15</span><span class="n">P0A0YFFRD6</span>       <span class="o">|</span> <span class="n">Good</span>
</pre></div>
</div>
</div>
<p>After you have enabled libstoragemgmt support, the output will look something
like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># ceph orch device ls</span>
<span class="n">Hostname</span>   <span class="n">Path</span>      <span class="n">Type</span>  <span class="n">Serial</span>              <span class="n">Size</span>   <span class="n">Health</span>   <span class="n">Ident</span>  <span class="n">Fault</span>  <span class="n">Available</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>     <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">P0A0YFFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Good</span>     <span class="n">Off</span>    <span class="n">Off</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>     <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A08WFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Good</span>     <span class="n">Off</span>    <span class="n">Off</span>    <span class="n">No</span>
<span class="p">:</span>
</pre></div>
</div>
<p>In this example, libstoragemgmt has confirmed the health of the drives and the ability to
interact with the Identification and Fault LEDs on the drive enclosures. For further
information about interacting with these LEDs, refer to <a class="reference external" href="../rados/operations/devices">device management</a>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The current release of <a class="reference external" href="https://github.com/libstorage/libstoragemgmt">libstoragemgmt</a> (1.8.8) supports SCSI, SAS, and SATA based
local disks only. There is no official support for NVMe devices (PCIe)</p>
</div>
</div>
<div class="section" id="deploy-osds">
<span id="cephadm-deploy-osds"></span><h2>Deploy OSDs<a class="headerlink" href="#deploy-osds" title="Permalink to this headline">¶</a></h2>
<div class="section" id="listing-storage-devices">
<h3>Listing Storage Devices<a class="headerlink" href="#listing-storage-devices" title="Permalink to this headline">¶</a></h3>
<p>In order to deploy an OSD, there must be a storage device that is <em>available</em> on
which the OSD will be deployed.</p>
<p>Run this command to display an inventory of storage devices on all cluster hosts:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch device ls</span>
</pre></div></div><p>A storage device is considered <em>available</em> if all of the following
conditions are met:</p>
<ul class="simple">
<li><p>The device must have no partitions.</p></li>
<li><p>The device must not have any LVM state.</p></li>
<li><p>The device must not be mounted.</p></li>
<li><p>The device must not contain a file system.</p></li>
<li><p>The device must not contain a Ceph BlueStore OSD.</p></li>
<li><p>The device must be larger than 5 GB.</p></li>
</ul>
<p>Ceph will not provision an OSD on a device that is not available.</p>
</div>
<div class="section" id="creating-new-osds">
<h3>Creating New OSDs<a class="headerlink" href="#creating-new-osds" title="Permalink to this headline">¶</a></h3>
<p>There are a few ways to create new OSDs:</p>
<ul>
<li><p>Tell Ceph to consume any available and unused storage device:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch apply osd --all-available-devices</span>
</pre></div></div></li>
<li><p>Create an OSD from a specific device on a specific host:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch daemon add osd *&lt;host&gt;*:*&lt;device-path&gt;*</span>
</pre></div></div><p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch daemon add osd host1:/dev/sdb</span>
</pre></div></div></li>
<li><p>You can use <a class="reference internal" href="#drivegroups"><span class="std std-ref">Advanced OSD Service Specifications</span></a> to categorize device(s) based on their
properties. This might be useful in forming a clearer picture of which
devices are available to consume. Properties include device type (SSD or
HDD), device model names, size, and the hosts on which the devices exist:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch apply -i spec.yml</span>
</pre></div></div></li>
</ul>
</div>
<div class="section" id="dry-run">
<h3>Dry Run<a class="headerlink" href="#dry-run" title="Permalink to this headline">¶</a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag causes the orchestrator to present a preview of what
will happen without actually creating the OSDs.</p>
<p>For example:</p>
<blockquote>
<div><div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch apply osd --all-available-devices --dry-run</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">NAME</span>                  <span class="n">HOST</span>  <span class="n">DATA</span>      <span class="n">DB</span>  <span class="n">WAL</span>
<span class="nb">all</span><span class="o">-</span><span class="n">available</span><span class="o">-</span><span class="n">devices</span> <span class="n">node1</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdb</span>  <span class="o">-</span>   <span class="o">-</span>
<span class="nb">all</span><span class="o">-</span><span class="n">available</span><span class="o">-</span><span class="n">devices</span> <span class="n">node2</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdc</span>  <span class="o">-</span>   <span class="o">-</span>
<span class="nb">all</span><span class="o">-</span><span class="n">available</span><span class="o">-</span><span class="n">devices</span> <span class="n">node3</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdd</span>  <span class="o">-</span>   <span class="o">-</span>
</pre></div>
</div>
</div></blockquote>
</div>
<div class="section" id="declarative-state">
<span id="cephadm-osd-declarative"></span><h3>Declarative State<a class="headerlink" href="#declarative-state" title="Permalink to this headline">¶</a></h3>
<p>The effect of <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code> is persistent. This means that drives that
are added to the system after the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code> command completes will be
automatically found and added to the cluster.  It also means that drives that
become available (by zapping, for example) after the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code>
command completes will be automatically found and added to the cluster.</p>
<p>We will examine the effects of the following command:</p>
<blockquote>
<div><div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch apply osd --all-available-devices</span>
</pre></div></div></div></blockquote>
<p>After running the above command:</p>
<ul class="simple">
<li><p>If you add new disks to the cluster, they will automatically be used to
create new OSDs.</p></li>
<li><p>If you remove an OSD and clean the LVM physical volume, a new OSD will be
created automatically.</p></li>
</ul>
<p>To disable the automatic creation of OSD on available devices, use the
<code class="docutils literal notranslate"><span class="pre">unmanaged</span></code> parameter:</p>
<p>If you want to avoid this behavior (disable automatic creation of OSD on available devices), use the <code class="docutils literal notranslate"><span class="pre">unmanaged</span></code> parameter:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch apply osd --all-available-devices --unmanaged<span class="o">=</span><span class="nb">true</span></span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>Keep these three facts in mind:</p>
<ul class="simple">
<li><p>The default behavior of <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code> causes cephadm constantly to reconcile. This means that cephadm creates OSDs as soon as new drives are detected.</p></li>
<li><p>Setting <code class="docutils literal notranslate"><span class="pre">unmanaged:</span> <span class="pre">True</span></code> disables the creation of OSDs. If <code class="docutils literal notranslate"><span class="pre">unmanaged:</span> <span class="pre">True</span></code> is set, nothing will happen even if you apply a new OSD service.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">daemon</span> <span class="pre">add</span></code> creates OSDs, but does not add an OSD service.</p></li>
</ul>
</div>
<ul class="simple">
<li><p>For cephadm, see also <a class="reference internal" href="../#cephadm-spec-unmanaged"><span class="std std-ref">Disabling automatic deployment of daemons</span></a>.</p></li>
</ul>
</div>
</div>
<div class="section" id="remove-an-osd">
<span id="cephadm-osd-removal"></span><h2>Remove an OSD<a class="headerlink" href="#remove-an-osd" title="Permalink to this headline">¶</a></h2>
<p>Removing an OSD from a cluster involves two steps:</p>
<ol class="arabic simple">
<li><p>evacuating all placement groups (PGs) from the cluster</p></li>
<li><p>removing the PG-free OSD from the cluster</p></li>
</ol>
<p>The following command performs these two steps:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch osd rm &lt;osd_id<span class="o">(</span>s<span class="o">)</span>&gt; <span class="o">[</span>--replace<span class="o">]</span> <span class="o">[</span>--force<span class="o">]</span></span>
</pre></div></div><p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch osd rm <span class="m">0</span></span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Scheduled</span> <span class="n">OSD</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">removal</span>
</pre></div>
</div>
<p>OSDs that are not safe to destroy will be rejected.</p>
<div class="section" id="monitoring-osd-state">
<h3>Monitoring OSD State<a class="headerlink" href="#monitoring-osd-state" title="Permalink to this headline">¶</a></h3>
<p>You can query the state of OSD operation with the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch osd rm status</span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">OSD_ID</span>  <span class="n">HOST</span>         <span class="n">STATE</span>                    <span class="n">PG_COUNT</span>  <span class="n">REPLACE</span>  <span class="n">FORCE</span>  <span class="n">STARTED_AT</span>
<span class="mi">2</span>       <span class="n">cephadm</span><span class="o">-</span><span class="n">dev</span>  <span class="n">done</span><span class="p">,</span> <span class="n">waiting</span> <span class="k">for</span> <span class="n">purge</span>  <span class="mi">0</span>         <span class="kc">True</span>     <span class="kc">False</span>  <span class="mi">2020</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">17</span> <span class="mi">13</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">43.147684</span>
<span class="mi">3</span>       <span class="n">cephadm</span><span class="o">-</span><span class="n">dev</span>  <span class="n">draining</span>                 <span class="mi">17</span>        <span class="kc">False</span>    <span class="kc">True</span>   <span class="mi">2020</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">17</span> <span class="mi">13</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">45.162158</span>
<span class="mi">4</span>       <span class="n">cephadm</span><span class="o">-</span><span class="n">dev</span>  <span class="n">started</span>                  <span class="mi">42</span>        <span class="kc">False</span>    <span class="kc">True</span>   <span class="mi">2020</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">17</span> <span class="mi">13</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">45.162158</span>
</pre></div>
</div>
<p>When no PGs are left on the OSD, it will be decommissioned and removed from the cluster.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>After removing an OSD, if you wipe the LVM physical volume in the device used by the removed OSD, a new OSD will be created.
For more information on this, read about the <code class="docutils literal notranslate"><span class="pre">unmanaged</span></code> parameter in <a class="reference internal" href="#cephadm-osd-declarative"><span class="std std-ref">Declarative State</span></a>.</p>
</div>
</div>
<div class="section" id="stopping-osd-removal">
<h3>Stopping OSD Removal<a class="headerlink" href="#stopping-osd-removal" title="Permalink to this headline">¶</a></h3>
<p>It is possible to stop queued OSD removals by using the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch osd rm stop &lt;osd_id<span class="o">(</span>s<span class="o">)</span>&gt;</span>
</pre></div></div><p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch osd rm stop <span class="m">4</span></span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Stopped</span> <span class="n">OSD</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="n">removal</span>
</pre></div>
</div>
<p>This resets the initial state of the OSD and takes it off the removal queue.</p>
</div>
<div class="section" id="replacing-an-osd">
<h3>Replacing an OSD<a class="headerlink" href="#replacing-an-osd" title="Permalink to this headline">¶</a></h3>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">orch osd rm &lt;osd_id<span class="o">(</span>s<span class="o">)</span>&gt; --replace <span class="o">[</span>--force<span class="o">]</span></span>
</pre></div></div><p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch osd rm <span class="m">4</span> --replace</span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Scheduled</span> <span class="n">OSD</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">replacement</span>
</pre></div>
</div>
<p>This follows the same procedure as the procedure in the “Remove OSD” section, with
one exception: the OSD is not permanently removed from the CRUSH hierarchy, but is
instead assigned a ‘destroyed’ flag.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The new OSD that will replace the removed OSD must be created on the same host
as the OSD that was removed.</p>
</div>
<p><strong>Preserving the OSD ID</strong></p>
<p>The ‘destroyed’ flag is used to determine which OSD ids will be reused in the
next OSD deployment.</p>
<p>If you use OSDSpecs for OSD deployment, your newly added disks will be assigned
the OSD ids of their replaced counterparts. This assumes that the new disks
still match the OSDSpecs.</p>
<p>Use the <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag to make certain that the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span> <span class="pre">osd</span></code>
command does what you want it to. The <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag shows you what the
outcome of the command will be without making the changes you specify. When
you are satisfied that the command will do what you want, run the command
without the <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag.</p>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>The name of your OSDSpec can be retrieved with the command <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">ls</span></code></p>
</div>
<p>Alternatively, you can use your OSDSpec file:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch apply -i &lt;osd_spec_file&gt; --dry-run</span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">NAME</span>                  <span class="n">HOST</span>  <span class="n">DATA</span>     <span class="n">DB</span> <span class="n">WAL</span>
<span class="o">&lt;</span><span class="n">name_of_osd_spec</span><span class="o">&gt;</span>    <span class="n">node1</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdb</span> <span class="o">-</span>  <span class="o">-</span>
</pre></div>
</div>
<p>When this output reflects your intention, omit the <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag to
execute the deployment.</p>
</div>
<div class="section" id="erasing-devices-zapping-devices">
<h3>Erasing Devices (Zapping Devices)<a class="headerlink" href="#erasing-devices-zapping-devices" title="Permalink to this headline">¶</a></h3>
<p>Erase (zap) a device so that it can be reused. <code class="docutils literal notranslate"><span class="pre">zap</span></code> calls <code class="docutils literal notranslate"><span class="pre">ceph-volume</span>
<span class="pre">zap</span></code> on the remote host.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch device zap &lt;hostname&gt; &lt;path&gt;</span>
</pre></div></div><p>Example command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph orch device zap my_hostname /dev/sdx</span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>If the unmanaged flag is unset, cephadm automatically deploys drives that
match the OSDSpec.  For example, if you use the
<code class="docutils literal notranslate"><span class="pre">all-available-devices</span></code> option when creating OSDs, when you <code class="docutils literal notranslate"><span class="pre">zap</span></code> a
device the cephadm orchestrator automatically creates a new OSD in the
device.  To disable this behavior, see <a class="reference internal" href="#cephadm-osd-declarative"><span class="std std-ref">Declarative State</span></a>.</p>
</div>
</div>
</div>
<div class="section" id="automatically-tuning-osd-memory">
<span id="osd-autotune"></span><h2>Automatically tuning OSD memory<a class="headerlink" href="#automatically-tuning-osd-memory" title="Permalink to this headline">¶</a></h2>
<p>OSD daemons will adjust their memory consumption based on the
<code class="docutils literal notranslate"><span class="pre">osd_memory_target</span></code> config option (several gigabytes, by
default).  If Ceph is deployed on dedicated nodes that are not sharing
memory with other services, cephadm can automatically adjust the per-OSD
memory consumption based on the total amount of RAM and the number of deployed
OSDs.</p>
<p>This option is enabled globally with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span> <span class="nb">set</span> <span class="n">osd</span> <span class="n">osd_memory_target_autotune</span> <span class="n">true</span>
</pre></div>
</div>
<p>Cephadm will start with a fraction
(<code class="docutils literal notranslate"><span class="pre">mgr/cephadm/autotune_memory_target_ratio</span></code>, which defaults to
<code class="docutils literal notranslate"><span class="pre">.7</span></code>) of the total RAM in the system, subtract off any memory
consumed by non-autotuned daemons (non-OSDs, for OSDs for which
<code class="docutils literal notranslate"><span class="pre">osd_memory_target_autotune</span></code> is false), and then divide by the
remaining OSDs.</p>
<p>The final targets are reflected in the config database with options like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">WHO</span>   <span class="n">MASK</span>      <span class="n">LEVEL</span>   <span class="n">OPTION</span>              <span class="n">VALUE</span>
<span class="n">osd</span>   <span class="n">host</span><span class="p">:</span><span class="n">foo</span>  <span class="n">basic</span>   <span class="n">osd_memory_target</span>   <span class="mi">126092301926</span>
<span class="n">osd</span>   <span class="n">host</span><span class="p">:</span><span class="n">bar</span>  <span class="n">basic</span>   <span class="n">osd_memory_target</span>   <span class="mi">6442450944</span>
</pre></div>
</div>
<p>Both the limits and the current memory consumed by each daemon are visible from
the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">ps</span></code> output in the <code class="docutils literal notranslate"><span class="pre">MEM</span> <span class="pre">LIMIT</span></code> column:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">NAME</span>        <span class="n">HOST</span>  <span class="n">PORTS</span>  <span class="n">STATUS</span>         <span class="n">REFRESHED</span>  <span class="n">AGE</span>  <span class="n">MEM</span> <span class="n">USED</span>  <span class="n">MEM</span> <span class="n">LIMIT</span>  <span class="n">VERSION</span>                <span class="n">IMAGE</span> <span class="n">ID</span>      <span class="n">CONTAINER</span> <span class="n">ID</span>
<span class="n">osd</span><span class="mf">.1</span>       <span class="n">dael</span>         <span class="n">running</span> <span class="p">(</span><span class="mi">3</span><span class="n">h</span><span class="p">)</span>     <span class="mi">10</span><span class="n">s</span> <span class="n">ago</span>   <span class="mi">3</span><span class="n">h</span>    <span class="mi">72857</span><span class="n">k</span>     <span class="mf">117.4</span><span class="n">G</span>  <span class="mf">17.0.0</span><span class="o">-</span><span class="mi">3781</span><span class="o">-</span><span class="n">gafaed750</span>  <span class="mi">7015</span><span class="n">fda3cd67</span>  <span class="mf">9e183363</span><span class="n">d39c</span>
<span class="n">osd</span><span class="mf">.2</span>       <span class="n">dael</span>         <span class="n">running</span> <span class="p">(</span><span class="mi">81</span><span class="n">m</span><span class="p">)</span>    <span class="mi">10</span><span class="n">s</span> <span class="n">ago</span>  <span class="mi">81</span><span class="n">m</span>    <span class="mi">63989</span><span class="n">k</span>     <span class="mf">117.4</span><span class="n">G</span>  <span class="mf">17.0.0</span><span class="o">-</span><span class="mi">3781</span><span class="o">-</span><span class="n">gafaed750</span>  <span class="mi">7015</span><span class="n">fda3cd67</span>  <span class="mi">1</span><span class="n">f0cc479b051</span>
<span class="n">osd</span><span class="mf">.3</span>       <span class="n">dael</span>         <span class="n">running</span> <span class="p">(</span><span class="mi">62</span><span class="n">m</span><span class="p">)</span>    <span class="mi">10</span><span class="n">s</span> <span class="n">ago</span>  <span class="mi">62</span><span class="n">m</span>    <span class="mi">64071</span><span class="n">k</span>     <span class="mf">117.4</span><span class="n">G</span>  <span class="mf">17.0.0</span><span class="o">-</span><span class="mi">3781</span><span class="o">-</span><span class="n">gafaed750</span>  <span class="mi">7015</span><span class="n">fda3cd67</span>  <span class="n">ac5537492f27</span>
</pre></div>
</div>
<p>To exclude an OSD from memory autotuning, disable the autotune option
for that OSD and also set a specific memory target.  For example,</p>
<blockquote>
<div><div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph config <span class="nb">set</span> osd.123 osd_memory_target_autotune <span class="nb">false</span></span>
<span class="prompt1">ceph config <span class="nb">set</span> osd.123 osd_memory_target 16G</span>
</pre></div></div></div></blockquote>
</div>
<div class="section" id="advanced-osd-service-specifications">
<span id="drivegroups"></span><h2>Advanced OSD Service Specifications<a class="headerlink" href="#advanced-osd-service-specifications" title="Permalink to this headline">¶</a></h2>
<p><a class="reference internal" href="../#orchestrator-cli-service-spec"><span class="std std-ref">Service Specification</span></a>s of type <code class="docutils literal notranslate"><span class="pre">osd</span></code> are a way to describe a
cluster layout, using the properties of disks. Service specifications give the
user an abstract way to tell Ceph which disks should turn into OSDs with which
configurations, without knowing the specifics of device names and paths.</p>
<p>Service specifications make it possible to define a yaml or json file that can
be used to reduce the amount of manual work involved in creating OSDs.</p>
<p>For example, instead of running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt2:before {
  content: "[monitor.1]# ";
}
</style><span class="prompt2">ceph orch daemon add osd *&lt;host&gt;*:*&lt;path-to-device&gt;*</span>
</pre></div></div><p>for each device and each host, we can define a yaml or json file that allows us
to describe the layout. Here’s the most basic example.</p>
<p>Create a file called (for example) <code class="docutils literal notranslate"><span class="pre">osd_spec.yml</span></code>:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">default_drive_group</span>  <span class="c1"># custom name of the osd spec</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>              <span class="c1"># which hosts to target</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>                  <span class="c1"># the type of devices you are applying specs to</span>
    <span class="nt">all</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">true</span>                    <span class="c1"># a filter, check below for a full list</span>
</pre></div>
</div>
<p>This means :</p>
<ol class="arabic">
<li><p>Turn any available device (ceph-volume decides what ‘available’ is) into an
OSD on all hosts that match the glob pattern ‘*’. (The glob pattern matches
against the registered hosts from <cite>host ls</cite>) A more detailed section on
host_pattern is available below.</p></li>
<li><p>Then pass it to <cite>osd create</cite> like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2">ceph orch apply -i /path/to/osd_spec.yml</span>
</pre></div></div><p>This instruction will be issued to all the matching hosts, and will deploy
these OSDs.</p>
<p>Setups more complex than the one specified by the <code class="docutils literal notranslate"><span class="pre">all</span></code> filter are
possible. See <a class="reference internal" href="#osd-filters"><span class="std std-ref">Filters</span></a> for details.</p>
<p>A <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag can be passed to the <code class="docutils literal notranslate"><span class="pre">apply</span> <span class="pre">osd</span></code> command to display a
synopsis of the proposed layout.</p>
</li>
</ol>
<p>Example</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2">ceph orch apply -i /path/to/osd_spec.yml --dry-run</span>
</pre></div></div><div class="section" id="filters">
<span id="osd-filters"></span><h3>Filters<a class="headerlink" href="#filters" title="Permalink to this headline">¶</a></h3>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Filters are applied using an <cite>AND</cite> gate by default. This means that a drive
must fulfill all filter criteria in order to get selected. This behavior can
be adjusted by setting <code class="docutils literal notranslate"><span class="pre">filter_logic:</span> <span class="pre">OR</span></code> in the OSD specification.</p>
</div>
<p>Filters are used to assign disks to groups, using their attributes to group
them.</p>
<p>The attributes are based off of ceph-volume’s disk query. You can retrieve
information about the attributes with this command:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>ceph-volume inventory &lt;/path/to/disk&gt;
</pre></div>
</div>
<div class="section" id="vendor-or-model">
<h4>Vendor or Model<a class="headerlink" href="#vendor-or-model" title="Permalink to this headline">¶</a></h4>
<p>Specific disks can be targeted by vendor or model:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">disk_model_name</span>
</pre></div>
</div>
<p>or</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">vendor</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">disk_vendor_name</span>
</pre></div>
</div>
</div>
<div class="section" id="size">
<h4>Size<a class="headerlink" href="#size" title="Permalink to this headline">¶</a></h4>
<p>Specific disks can be targeted by <cite>Size</cite>:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">size_spec</span>
</pre></div>
</div>
<div class="section" id="size-specs">
<h5>Size specs<a class="headerlink" href="#size-specs" title="Permalink to this headline">¶</a></h5>
<p>Size specifications can be of the following forms:</p>
<ul class="simple">
<li><p>LOW:HIGH</p></li>
<li><p>:HIGH</p></li>
<li><p>LOW:</p></li>
<li><p>EXACT</p></li>
</ul>
<p>Concrete examples:</p>
<p>To include disks of an exact size</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span> <span class="s">&#39;10G&#39;</span>
</pre></div>
</div>
<p>To include disks within a given range of size:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span> <span class="s">&#39;10G:40G&#39;</span>
</pre></div>
</div>
<p>To include disks that are less than or equal to 10G in size:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span> <span class="s">&#39;:10G&#39;</span>
</pre></div>
</div>
<p>To include disks equal to or greater than 40G in size:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span> <span class="s">&#39;40G:&#39;</span>
</pre></div>
</div>
<p>Sizes don’t have to be specified exclusively in Gigabytes(G).</p>
<p>Other units of size are supported: Megabyte(M), Gigabyte(G) and Terrabyte(T).
Appending the (B) for byte is also supported: <code class="docutils literal notranslate"><span class="pre">MB</span></code>, <code class="docutils literal notranslate"><span class="pre">GB</span></code>, <code class="docutils literal notranslate"><span class="pre">TB</span></code>.</p>
</div>
</div>
<div class="section" id="rotational">
<h4>Rotational<a class="headerlink" href="#rotational" title="Permalink to this headline">¶</a></h4>
<p>This operates on the ‘rotational’ attribute of the disk.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">rotational</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">0 | 1</span>
</pre></div>
</div>
<p><cite>1</cite> to match all disks that are rotational</p>
<p><cite>0</cite> to match all disks that are non-rotational (SSD, NVME etc)</p>
</div>
<div class="section" id="all">
<h4>All<a class="headerlink" href="#all" title="Permalink to this headline">¶</a></h4>
<p>This will take all disks that are ‘available’</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This is exclusive for the data_devices section.</p>
</div>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">all</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">true</span>
</pre></div>
</div>
</div>
<div class="section" id="limiter">
<h4>Limiter<a class="headerlink" href="#limiter" title="Permalink to this headline">¶</a></h4>
<p>If you have specified some valid filters but want to limit the number of disks that they match, use the <code class="docutils literal notranslate"><span class="pre">limit</span></code> directive:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">limit</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">2</span>
</pre></div>
</div>
<p>For example, if you used <cite>vendor</cite> to match all disks that are from <cite>VendorA</cite>
but want to use only the first two, you could use <cite>limit</cite>:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">data_devices</span><span class="p">:</span>
  <span class="nt">vendor</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">VendorA</span>
  <span class="nt">limit</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">2</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p><cite>limit</cite> is a last resort and shouldn’t be used if it can be avoided.</p>
</div>
</div>
</div>
<div class="section" id="additional-options">
<h3>Additional Options<a class="headerlink" href="#additional-options" title="Permalink to this headline">¶</a></h3>
<p>There are multiple optional settings you can use to change the way OSDs are deployed.
You can add these options to the base level of an OSD spec for it to take effect.</p>
<p>This example would deploy all OSDs with encryption enabled.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">example_osd_spec</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">all</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">true</span>
  <span class="nt">encrypted</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">true</span>
</pre></div>
</div>
<p>See a full list in the DriveGroupSpecs</p>
<dl class="py class">
<dt id="ceph.deployment.drive_group.DriveGroupSpec">
<em class="property"><span class="pre">class</span> </em><code class="sig-prename descclassname"><span class="pre">ceph.deployment.drive_group.</span></code><code class="sig-name descname"><span class="pre">DriveGroupSpec</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Any</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Any</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec" title="Permalink to this definition">¶</a></dt>
<dd><p>Describe a drive group in the same form that ceph-volume
understands.</p>
<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.block_db_size">
<code class="sig-name descname"><span class="pre">block_db_size</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">,</span> </span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.block_db_size" title="Permalink to this definition">¶</a></dt>
<dd><p>Set (or override) the “bluestore_block_db_size” value, in bytes</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.block_wal_size">
<code class="sig-name descname"><span class="pre">block_wal_size</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">,</span> </span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.block_wal_size" title="Permalink to this definition">¶</a></dt>
<dd><p>Set (or override) the “bluestore_block_wal_size” value, in bytes</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.data_allocate_fraction">
<code class="sig-name descname"><span class="pre">data_allocate_fraction</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.data_allocate_fraction" title="Permalink to this definition">¶</a></dt>
<dd><p>Allocate a fraction of the data device (0,1.0]</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.data_devices">
<code class="sig-name descname"><span class="pre">data_devices</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.data_devices" title="Permalink to this definition">¶</a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.data_directories">
<code class="sig-name descname"><span class="pre">data_directories</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.data_directories" title="Permalink to this definition">¶</a></dt>
<dd><p>A list of strings, containing paths which should back OSDs</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.db_devices">
<code class="sig-name descname"><span class="pre">db_devices</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.db_devices" title="Permalink to this definition">¶</a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.db_slots">
<code class="sig-name descname"><span class="pre">db_slots</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.db_slots" title="Permalink to this definition">¶</a></dt>
<dd><p>How many OSDs per DB device</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.encrypted">
<code class="sig-name descname"><span class="pre">encrypted</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.encrypted" title="Permalink to this definition">¶</a></dt>
<dd><p><code class="docutils literal notranslate"><span class="pre">true</span></code> or <code class="docutils literal notranslate"><span class="pre">false</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.filter_logic">
<code class="sig-name descname"><span class="pre">filter_logic</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.filter_logic" title="Permalink to this definition">¶</a></dt>
<dd><p>The logic gate we use to match disks with filters.
defaults to ‘AND’</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.journal_devices">
<code class="sig-name descname"><span class="pre">journal_devices</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.journal_devices" title="Permalink to this definition">¶</a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.journal_size">
<code class="sig-name descname"><span class="pre">journal_size</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">,</span> </span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.journal_size" title="Permalink to this definition">¶</a></dt>
<dd><p>set journal_size in bytes</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.objectstore">
<code class="sig-name descname"><span class="pre">objectstore</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.objectstore" title="Permalink to this definition">¶</a></dt>
<dd><p><code class="docutils literal notranslate"><span class="pre">filestore</span></code> or <code class="docutils literal notranslate"><span class="pre">bluestore</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.osd_id_claims">
<code class="sig-name descname"><span class="pre">osd_id_claims</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.osd_id_claims" title="Permalink to this definition">¶</a></dt>
<dd><p>Optional: mapping of host -&gt; List of osd_ids that should be replaced
See <a class="reference internal" href="../../../mgr/orchestrator_modules/#orchestrator-osd-replace"><span class="std std-ref">OSD 替换</span></a></p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.osds_per_device">
<code class="sig-name descname"><span class="pre">osds_per_device</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.osds_per_device" title="Permalink to this definition">¶</a></dt>
<dd><p>Number of osd daemons per “DATA” device.
To fully utilize nvme devices multiple osds are required.</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.preview_only">
<code class="sig-name descname"><span class="pre">preview_only</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.preview_only" title="Permalink to this definition">¶</a></dt>
<dd><p>If this should be treated as a ‘preview’ spec</p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.wal_devices">
<code class="sig-name descname"><span class="pre">wal_devices</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.wal_devices" title="Permalink to this definition">¶</a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt id="ceph.deployment.drive_group.DriveGroupSpec.wal_slots">
<code class="sig-name descname"><span class="pre">wal_slots</span></code><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.wal_slots" title="Permalink to this definition">¶</a></dt>
<dd><p>How many OSDs per WAL device</p>
</dd></dl>

</dd></dl>

</div>
</div>
<div class="section" id="examples">
<h2>Examples<a class="headerlink" href="#examples" title="Permalink to this headline">¶</a></h2>
<div class="section" id="the-simple-case">
<h3>The simple case<a class="headerlink" href="#the-simple-case" title="Permalink to this headline">¶</a></h3>
<p>All nodes with the same setup</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>20 HDDs
Vendor: VendorA
Model: HDD-123-foo
Size: 4TB

2 SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB
</pre></div>
</div>
<p>This is a common setup and can be described quite easily:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">HDD-123-foo</span> <span class="c1"># Note, HDD-123 would also be valid</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span> <span class="c1"># Same here, MC-55-44 is valid</span>
</pre></div>
</div>
<p>However, we can improve it by reducing the filters on core properties of the drives:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">rotational</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">1</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">rotational</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">0</span>
</pre></div>
</div>
<p>Now, we enforce all rotating devices to be declared as ‘data devices’ and all non-rotating devices will be used as shared_devices (wal, db)</p>
<p>If you know that drives with more than 2TB will always be the slower data devices, you can also filter by size:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">size</span><span class="p">:</span> <span class="s">&#39;2TB:&#39;</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">size</span><span class="p">:</span> <span class="s">&#39;:2TB&#39;</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>All of the above OSD specs  are equally valid. Which of those you want to use depends on taste and on how much you expect your node layout to change.</p>
</div>
</div>
<div class="section" id="multiple-osd-specs-for-a-single-host">
<h3>Multiple OSD specs for a single host<a class="headerlink" href="#multiple-osd-specs-for-a-single-host" title="Permalink to this headline">¶</a></h3>
<p>Here we have two distinct setups</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>20 HDDs
Vendor: VendorA
Model: HDD-123-foo
Size: 4TB

12 SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB

2 NVMEs
Vendor: VendorC
Model: NVME-QQQQ-987
Size: 256GB
</pre></div>
</div>
<ul class="simple">
<li><p>20 HDDs should share 2 SSDs</p></li>
<li><p>10 SSDs should share 2 NVMes</p></li>
</ul>
<p>This can be described with two layouts.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd_spec_hdd</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">rotational</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">0</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span>
    <span class="nt">limit</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">2</span> <span class="c1"># db_slots is actually to be favoured here, but it&#39;s not implemented yet</span>
<span class="nn">---</span>
<span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd_spec_ssd</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">vendor</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">VendorC</span>
</pre></div>
</div>
<p>This would create the desired layout by using all HDDs as data_devices with two SSD assigned as dedicated db/wal devices.
The remaining SSDs(8) will be data_devices that have the ‘VendorC’ NVMEs assigned as dedicated db/wal devices.</p>
</div>
<div class="section" id="multiple-hosts-with-the-same-disk-layout">
<h3>Multiple hosts with the same disk layout<a class="headerlink" href="#multiple-hosts-with-the-same-disk-layout" title="Permalink to this headline">¶</a></h3>
<p>Assuming the cluster has different kinds of hosts each with similar disk
layout, it is recommended to apply different OSD specs matching only one
set of hosts. Typically you will have a spec for multiple hosts with the
same layout.</p>
<p>The sevice id as the unique key: In case a new OSD spec with an already
applied service id is applied, the existing OSD spec will be superseeded.
cephadm will now create new OSD daemons based on the new spec
definition. Existing OSD daemons will not be affected. See <a class="reference internal" href="#cephadm-osd-declarative"><span class="std std-ref">Declarative State</span></a>.</p>
<p>Node1-5</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>20 HDDs
Vendor: Intel
Model: SSD-123-foo
Size: 4TB
2 SSDs
Vendor: VendorA
Model: MC-55-44-ZX
Size: 512GB
</pre></div>
</div>
<p>Node6-10</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>5 NVMEs
Vendor: Intel
Model: SSD-123-foo
Size: 4TB
20 SSDs
Vendor: VendorA
Model: MC-55-44-ZX
Size: 512GB
</pre></div>
</div>
<p>You can use the ‘placement’ key in the layout to target certain nodes.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">disk_layout_a</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">label</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">disk_layout_a</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">rotational</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">1</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">rotational</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">0</span>
<span class="nn">---</span>
<span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">disk_layout_b</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">label</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">disk_layout_b</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">SSD-123-foo</span>
</pre></div>
</div>
<p>This applies different OSD specs to different hosts depending on the <cite>placement</cite> key.
See <a class="reference internal" href="../#orchestrator-cli-placement-spec"><span class="std std-ref">Daemon Placement</span></a></p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Assuming each host has a unique disk layout, each OSD
spec needs to have a different service id</p>
</div>
</div>
<div class="section" id="dedicated-wal-db">
<h3>Dedicated wal + db<a class="headerlink" href="#dedicated-wal-db" title="Permalink to this headline">¶</a></h3>
<p>All previous cases co-located the WALs with the DBs.
It’s however possible to deploy the WAL on a dedicated device as well, if it makes sense.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>20 HDDs
Vendor: VendorA
Model: SSD-123-foo
Size: 4TB

2 SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB

2 NVMEs
Vendor: VendorC
Model: NVME-QQQQ-987
Size: 256GB
</pre></div>
</div>
<p>The OSD spec for this case would look like the following (using the <cite>model</cite> filter):</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">host_pattern</span><span class="p">:</span> <span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">SSD-123-foo</span>
  <span class="nt">wal_devices</span><span class="p">:</span>
    <span class="nt">model</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">NVME-QQQQ-987</span>
</pre></div>
</div>
<p>It is also possible to specify directly device paths in specific hosts like the following:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">osd_using_paths</span>
<span class="nt">placement</span><span class="p">:</span>
  <span class="nt">hosts</span><span class="p">:</span>
    <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">Node01</span>
    <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">Node02</span>
<span class="nt">spec</span><span class="p">:</span>
  <span class="nt">data_devices</span><span class="p">:</span>
    <span class="nt">paths</span><span class="p">:</span>
    <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">/dev/sdb</span>
  <span class="nt">db_devices</span><span class="p">:</span>
    <span class="nt">paths</span><span class="p">:</span>
    <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">/dev/sdc</span>
  <span class="nt">wal_devices</span><span class="p">:</span>
    <span class="nt">paths</span><span class="p">:</span>
    <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">/dev/sdd</span>
</pre></div>
</div>
<p>This can easily be done with other filters, like <cite>size</cite> or <cite>vendor</cite> as well.</p>
</div>
</div>
<div class="section" id="activate-existing-osds">
<span id="cephadm-osd-activate"></span><h2>Activate existing OSDs<a class="headerlink" href="#activate-existing-osds" title="Permalink to this headline">¶</a></h2>
<p>In case the OS of a host was reinstalled, existing OSDs need to be activated
again. For this use case, cephadm provides a wrapper for <a class="reference internal" href="../../../ceph-volume/lvm/activate/#ceph-volume-lvm-activate"><span class="std std-ref">activate</span></a> that
activates all existing OSDs on a host.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph cephadm osd activate &lt;host&gt;...</span>
</pre></div></div><p>This will scan all existing disks for OSDs and deploy corresponding daemons.</p>
</div>
<div class="section" id="futher-reading">
<h2>Futher Reading<a class="headerlink" href="#futher-reading" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li><p><a class="reference internal" href="../../../ceph-volume/#ceph-volume"><span class="std std-ref">ceph-volume</span></a></p></li>
<li><p><a class="reference internal" href="../../../rados/#rados-index"><span class="std std-ref">Ceph 存储集群</span></a></p></li>
</ul>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../rgw/" class="btn btn-neutral float-right" title="RGW Service" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../mgr/" class="btn btn-neutral float-left" title="MGR Service" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>